{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "653089c9",
   "metadata": {},
   "source": [
    "# Chapter 14: Building and Training A Music Transformer\n",
    "\n",
    "This chapter covers\n",
    "\n",
    "* Performance-based music representation through control messages and velocity values\n",
    "* Tokenize a piece of music and convert it to a sequence of indexes\n",
    "* Building and training a music Transformer \n",
    "* Generating a sequence of music events using the trained music Transformer\n",
    "* Converting a sequence of music events back to a MIDI file, to be played on a computer\n",
    "\n",
    "Sad that your favorite musician is no longer with us? Sad no more: Generative AI can bring them back to the stage!\n",
    "Take, for example, Layered Reality, a London-based company that's working on a project called Elvis Evolution.  The goal? To resurrect the legendary Elvis Presley using artificial intelligence (AI). By feeding a vast array of Elvis' official archival material, including video clips, photographs, and music, into a sophisticated computer model, this AI Elvis learns to mimic his singing, speaking, dancing, and walking with remarkable resemblance. The result? A digital performance that captures the essence of the late King himself.\n",
    "\n",
    "The Elvis Evolution project is a shining example of the transformative impact of generative AI across various industries. In the previous chapter, you explored the use of MuseGAN to create music that could pass as authentic multi-track compositions. MuseGAN views a piece of music as a multi-dimensional object, similar to an image, and generates complete music pieces that resemble those in the training dataset. Both real and AI-generated music are then evaluated by a critic, which helps refine the AI-generated music until it's indistinguishable from the real thing.\n",
    "\n",
    "In this chapter, you'll take a different approach to AI music creation, treating it as a sequence of musical events. We'll apply techniques from text generation, as discussed in Chapters 11 and 12, to predict the next element in a sequence. Specifically, you'll develop a GPT-style model to predict the next musical event based on all previous events in the sequence. GPT-style Transformers are ideal for this task because of their scalability and the self-attention mechanism, which helps them capture long-range dependencies and understand context. This makes them highly effective for sequence prediction and generation across a wide range of content, including music. The music Transformer you create has 20.16 million parameters, large enough to capture the long-term relations of different notes in music pieces, but smaller enough to be trained in a reasonable amount of time. \n",
    "\n",
    "We’ll use the Maestro piano music from Google’s Magenta group as our training data. You’ll learn how to first convert a MIDI (Musical Instrument Digital Interface) file into a sequence of music notes, analogous to raw text data in natural language processing (NLP). You’ll then break the musical notes down into small pieces called music events, analogous to tokens in NLP. Since neural networks can only accept numerical inputs, you’ll map each unique event token to an index. With this, the music pieces in the training data are converted into sequences of indexes, ready to be fed into neural networks. \n",
    "\n",
    "To train the music Transformer to predict the next token based on the current token and all previous tokens in the sequence, we’ll create sequences of 2048 indexes as inputs (features x). We then shift the sequences one index to the right and use them as the outputs (targets y). We feed pairs of (x, y) to the music Transformer to train it. Once trained, we’ll use a short sequence of indexes as the prompt and feed it to the music Transformer to predict the next token, which is then appended to the prompt to form a new sequence. This new sequence is fed back into the model for further predictions, and this process is repeated until the sequence reaches a desired length.\n",
    "\n",
    "You’ll see that the trained music Transformer can generate lifelike music that mimics the style in the training dataset. Further, unlike the music generated in Chapter 13, you’ll learn to control the creativity of the music piece. You’ll achieve this by scaling the predicted logits with the temperature parameter, just as you did in earlier chapters when controlling the creativity of the generated text. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c268aecc",
   "metadata": {},
   "source": [
    "# 1\tIntroduction to music Transformer\n",
    "# 2\tTokenize music pieces"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "15923a35",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: pretty_midi in c:\\users\\hlliu2\\anaconda3\\envs\\gan\\lib\\site-packages (0.2.10)\n",
      "Collecting music21\n",
      "  Downloading music21-9.1.0-py3-none-any.whl.metadata (4.8 kB)\n",
      "Requirement already satisfied: numpy>=1.7.0 in c:\\users\\hlliu2\\anaconda3\\envs\\gan\\lib\\site-packages (from pretty_midi) (1.23.5)\n",
      "Requirement already satisfied: mido>=1.1.16 in c:\\users\\hlliu2\\anaconda3\\envs\\gan\\lib\\site-packages (from pretty_midi) (1.3.2)\n",
      "Requirement already satisfied: six in c:\\users\\hlliu2\\anaconda3\\envs\\gan\\lib\\site-packages (from pretty_midi) (1.16.0)\n",
      "Requirement already satisfied: chardet in c:\\users\\hlliu2\\anaconda3\\envs\\gan\\lib\\site-packages (from music21) (4.0.0)\n",
      "Requirement already satisfied: joblib in c:\\users\\hlliu2\\anaconda3\\envs\\gan\\lib\\site-packages (from music21) (1.2.0)\n",
      "Collecting jsonpickle (from music21)\n",
      "  Downloading jsonpickle-3.0.3-py3-none-any.whl.metadata (7.3 kB)\n",
      "Requirement already satisfied: matplotlib in c:\\users\\hlliu2\\anaconda3\\envs\\gan\\lib\\site-packages (from music21) (3.7.1)\n",
      "Requirement already satisfied: more-itertools in c:\\users\\hlliu2\\anaconda3\\envs\\gan\\lib\\site-packages (from music21) (8.12.0)\n",
      "Requirement already satisfied: requests in c:\\users\\hlliu2\\anaconda3\\envs\\gan\\lib\\site-packages (from music21) (2.31.0)\n",
      "Collecting webcolors>=1.5 (from music21)\n",
      "  Downloading webcolors-1.13-py3-none-any.whl.metadata (2.6 kB)\n",
      "Requirement already satisfied: packaging~=23.1 in c:\\users\\hlliu2\\anaconda3\\envs\\gan\\lib\\site-packages (from mido>=1.1.16->pretty_midi) (23.2)\n",
      "Requirement already satisfied: contourpy>=1.0.1 in c:\\users\\hlliu2\\anaconda3\\envs\\gan\\lib\\site-packages (from matplotlib->music21) (1.0.7)\n",
      "Requirement already satisfied: cycler>=0.10 in c:\\users\\hlliu2\\anaconda3\\envs\\gan\\lib\\site-packages (from matplotlib->music21) (0.11.0)\n",
      "Requirement already satisfied: fonttools>=4.22.0 in c:\\users\\hlliu2\\anaconda3\\envs\\gan\\lib\\site-packages (from matplotlib->music21) (4.39.4)\n",
      "Requirement already satisfied: kiwisolver>=1.0.1 in c:\\users\\hlliu2\\anaconda3\\envs\\gan\\lib\\site-packages (from matplotlib->music21) (1.4.4)\n",
      "Requirement already satisfied: pillow>=6.2.0 in c:\\users\\hlliu2\\anaconda3\\envs\\gan\\lib\\site-packages (from matplotlib->music21) (9.3.0)\n",
      "Requirement already satisfied: pyparsing>=2.3.1 in c:\\users\\hlliu2\\anaconda3\\envs\\gan\\lib\\site-packages (from matplotlib->music21) (3.0.9)\n",
      "Requirement already satisfied: python-dateutil>=2.7 in c:\\users\\hlliu2\\anaconda3\\envs\\gan\\lib\\site-packages (from matplotlib->music21) (2.8.2)\n",
      "Requirement already satisfied: charset-normalizer<4,>=2 in c:\\users\\hlliu2\\anaconda3\\envs\\gan\\lib\\site-packages (from requests->music21) (2.0.4)\n",
      "Requirement already satisfied: idna<4,>=2.5 in c:\\users\\hlliu2\\anaconda3\\envs\\gan\\lib\\site-packages (from requests->music21) (3.4)\n",
      "Requirement already satisfied: urllib3<3,>=1.21.1 in c:\\users\\hlliu2\\anaconda3\\envs\\gan\\lib\\site-packages (from requests->music21) (1.26.15)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in c:\\users\\hlliu2\\anaconda3\\envs\\gan\\lib\\site-packages (from requests->music21) (2023.7.22)\n",
      "Downloading music21-9.1.0-py3-none-any.whl (22.8 MB)\n",
      "   ---------------------------------------- 0.0/22.8 MB ? eta -:--:--\n",
      "   ---------------------------------------- 0.2/22.8 MB 4.6 MB/s eta 0:00:05\n",
      "   - -------------------------------------- 1.1/22.8 MB 14.2 MB/s eta 0:00:02\n",
      "   ----- ---------------------------------- 3.4/22.8 MB 31.0 MB/s eta 0:00:01\n",
      "   ------- -------------------------------- 4.1/22.8 MB 24.1 MB/s eta 0:00:01\n",
      "   --------- ------------------------------ 5.3/22.8 MB 24.3 MB/s eta 0:00:01\n",
      "   ----------------- ---------------------- 9.9/22.8 MB 37.1 MB/s eta 0:00:01\n",
      "   -------------------------- ------------- 15.1/22.8 MB 81.8 MB/s eta 0:00:01\n",
      "   --------------------------------- ------ 19.0/22.8 MB 93.9 MB/s eta 0:00:01\n",
      "   -------------------------------------- - 22.0/22.8 MB 93.9 MB/s eta 0:00:01\n",
      "   ---------------------------------------- 22.8/22.8 MB 65.5 MB/s eta 0:00:00\n",
      "Downloading webcolors-1.13-py3-none-any.whl (14 kB)\n",
      "Downloading jsonpickle-3.0.3-py3-none-any.whl (40 kB)\n",
      "   ---------------------------------------- 0.0/40.8 kB ? eta -:--:--\n",
      "   ---------------------------------------- 40.8/40.8 kB ? eta 0:00:00\n",
      "Installing collected packages: webcolors, jsonpickle, music21\n",
      "Successfully installed jsonpickle-3.0.3 music21-9.1.0 webcolors-1.13\n"
     ]
    }
   ],
   "source": [
    "!pip install pretty_midi music21"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ed84eadd",
   "metadata": {},
   "source": [
    "## 2.1. Download MIDI Files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "c57af8c2",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "os.makedirs(\"files/maestro-v2.0.0/train\", exist_ok=True)\n",
    "os.makedirs(\"files/maestro-v2.0.0/val\", exist_ok=True)\n",
    "os.makedirs(\"files/maestro-v2.0.0/test\", exist_ok=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "4d463840",
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "import pickle\n",
    "from utils.processor import encode_midi\n",
    "\n",
    "file=\"files/maestro-v2.0.0/maestro-v2.0.0.json\"\n",
    "\n",
    "with open(file,\"r\") as fb:\n",
    "    maestro_json=json.load(fb)\n",
    "\n",
    "for x in maestro_json:\n",
    "    mid=rf'files/maestro-v2.0.0/{x[\"midi_filename\"]}'\n",
    "    split_type = x[\"split\"]\n",
    "    f_name = mid.split(\"/\")[-1] + \".pickle\"\n",
    "    if(split_type == \"train\"):\n",
    "        o_file = rf'files/maestro-v2.0.0/train/{f_name}'\n",
    "    elif(split_type == \"validation\"):\n",
    "        o_file = rf'files/maestro-v2.0.0/val/{f_name}'\n",
    "    elif(split_type == \"test\"):\n",
    "        o_file = rf'files/maestro-v2.0.0/test/{f_name}'\n",
    "    prepped = encode_midi(mid)\n",
    "    with open(o_file,\"wb\") as f:\n",
    "        pickle.dump(prepped, f)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "0db886ff",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "there are 967 files in the train set\n",
      "there are 137 files in the validation set\n",
      "there are 178 files in the test set\n"
     ]
    }
   ],
   "source": [
    "train_size=len(os.listdir('files/maestro-v2.0.0/train'))\n",
    "print(f\"there are {train_size} files in the train set\")\n",
    "val_size=len(os.listdir('files/maestro-v2.0.0/val'))\n",
    "print(f\"there are {val_size} files in the validation set\")\n",
    "test_size=len(os.listdir('files/maestro-v2.0.0/test'))\n",
    "print(f\"there are {test_size} files in the test set\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "55202435",
   "metadata": {},
   "source": [
    "## 2.2\tTokenize MIDI files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "04cfdab3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<[SNote] time: 1.0325520833333333 type: note_on, value: 74, velocity: 86>\n",
      "<[SNote] time: 1.0442708333333333 type: note_on, value: 38, velocity: 77>\n",
      "<[SNote] time: 1.2265625 type: note_off, value: 74, velocity: None>\n",
      "<[SNote] time: 1.2395833333333333 type: note_on, value: 73, velocity: 69>\n",
      "<[SNote] time: 1.2408854166666665 type: note_on, value: 37, velocity: 64>\n"
     ]
    }
   ],
   "source": [
    "import pickle\n",
    "from utils.processor import encode_midi\n",
    "import pretty_midi\n",
    "from utils.processor import (_control_preprocess,\n",
    "    _note_preprocess,_divide_note,\n",
    "    _make_time_sift_events,_snote2events)\n",
    "\n",
    "file='MIDI-Unprocessed_Chamber1_MID--AUDIO_07_R3_2018_wav--2'\n",
    "name=rf'files/maestro-v2.0.0/2018/{file}.midi'\n",
    "\n",
    "# encode\n",
    "events=[]\n",
    "notes=[]\n",
    "\n",
    "# convert song to an easily-manipulable format\n",
    "song=pretty_midi.PrettyMIDI(name)\n",
    "for inst in song.instruments:\n",
    "    inst_notes=inst.notes\n",
    "    ctrls=_control_preprocess([ctrl for ctrl in \n",
    "       inst.control_changes if ctrl.number == 64])\n",
    "    notes += _note_preprocess(ctrls, inst_notes)\n",
    "dnotes = _divide_note(notes)    \n",
    "dnotes.sort(key=lambda x: x.time)    \n",
    "for i in range(5):\n",
    "    print(dnotes[i])   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "34f2d3b0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<Event type: time_shift, value: 99>\n",
      "<Event type: time_shift, value: 2>\n",
      "<Event type: velocity, value: 21>\n",
      "<Event type: note_on, value: 74>\n",
      "<Event type: time_shift, value: 0>\n",
      "<Event type: velocity, value: 19>\n",
      "<Event type: note_on, value: 38>\n",
      "<Event type: time_shift, value: 17>\n",
      "<Event type: note_off, value: 74>\n",
      "<Event type: time_shift, value: 0>\n",
      "<Event type: velocity, value: 17>\n",
      "<Event type: note_on, value: 73>\n",
      "<Event type: velocity, value: 16>\n",
      "<Event type: note_on, value: 37>\n",
      "<Event type: time_shift, value: 0>\n"
     ]
    }
   ],
   "source": [
    "cur_time = 0\n",
    "cur_vel = 0\n",
    "for snote in dnotes:\n",
    "    events += _make_time_sift_events(prev_time=cur_time,\n",
    "                                     post_time=snote.time)\n",
    "    events += _snote2events(snote=snote, prev_vel=cur_vel)\n",
    "    cur_time = snote.time\n",
    "    cur_vel = snote.velocity    \n",
    "indexes=[e.to_int() for e in events]   \n",
    "for i in range(15):\n",
    "    print(events[i])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a57f798b",
   "metadata": {},
   "source": [
    "## 2.3\tPrepare the training data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "d1a83d38",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch,os,pickle\n",
    "\n",
    "max_seq=2048\n",
    "def create_xys(folder):  \n",
    "    files=[os.path.join(folder,f) for f in os.listdir(folder)]\n",
    "    xys=[]\n",
    "    for f in files:\n",
    "        with open(f,\"rb\") as fb:\n",
    "            music=pickle.load(fb)\n",
    "        music=torch.LongTensor(music)      \n",
    "        x=torch.full((max_seq,),389, dtype=torch.long)\n",
    "        y=torch.full((max_seq,),389, dtype=torch.long)\n",
    "        length=len(music)\n",
    "        if length<=max_seq:\n",
    "            print(length)\n",
    "            x[:length]=music\n",
    "            y[:length-1]=music[1:]\n",
    "            y[length-1]=388    \n",
    "        else:\n",
    "            x=music[:max_seq]\n",
    "            y=music[1:max_seq+1]   \n",
    "        xys.append((x,y))\n",
    "    return xys"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "400c351c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "15\n",
      "5\n",
      "1643\n",
      "1771\n",
      "586\n"
     ]
    }
   ],
   "source": [
    "trainfolder='files/maestro-v2.0.0/train'\n",
    "train=create_xys(trainfolder)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "068bf94f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "processing the validation set\n",
      "processing the test set\n",
      "1837\n"
     ]
    }
   ],
   "source": [
    "valfolder='files/maestro-v2.0.0/val'\n",
    "testfolder='files/maestro-v2.0.0/test'\n",
    "print(\"processing the validation set\")\n",
    "val=create_xys(valfolder)\n",
    "print(\"processing the test set\")\n",
    "test=create_xys(testfolder)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "c4823d8c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([2048])\n",
      "tensor([324, 366,  67,  ...,  60, 264, 369])\n"
     ]
    }
   ],
   "source": [
    "val1, _ = val[0]\n",
    "print(val1.shape)\n",
    "print(val1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "c738e99f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<pretty_midi.pretty_midi.PrettyMIDI at 0x12f04485510>"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from utils.processor import decode_midi\n",
    "\n",
    "file_path=\"files/val1.midi\"\n",
    "decode_midi(val1.cpu().numpy(), file_path=file_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "01667af8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <div id=\"midiPlayerDiv9268\"></div>\n",
       "        <link rel=\"stylesheet\" href=\"https://cuthbertLab.github.io/music21j/css/m21.css\">\n",
       "        \n",
       "        <script\n",
       "        src=\"https://cdnjs.cloudflare.com/ajax/libs/require.js/2.3.6/require.min.js\"\n",
       "        ></script>\n",
       "    \n",
       "        <script>\n",
       "        function midiPlayerDiv9268_play() {\n",
       "            const rq = require.config({\n",
       "                paths: {\n",
       "                    'music21': 'https://cuthbertLab.github.io/music21j/releases/music21.debug',\n",
       "                }\n",
       "            });\n",
       "            rq(['music21'], function(music21) {\n",
       "                mp = new music21.miditools.MidiPlayer();\n",
       "                mp.addPlayer(\"#midiPlayerDiv9268\");\n",
       "                mp.base64Load(\"data:audio/midi;base64,TVRoZAAAAAYAAQACJ2BNVHJrAAAAFAD/UQMHoSAA/1gEBAIYCM5g/y8ATVRyawAAEU0A/wMZRGV2ZWxvcGVkIEJ5IFlhbmctS2ljaGFuZwDAAQDgAEAAwAHiOJBKRACQNSigaJA+MIZIgEoAAIA1AACQQTwAkEVEjRCAPgCgaJBIQACQNyiNEIBBAACARQAAkEZEAJA5II0QgEgAAIA3AIZIkD00k1iQQDwAkEVIhkiAPQCgaIBGAI0QkEpMAJA1MIZIgEAAAIBFAJNYkD4wjRCAOQAAgEoAAIA1AIZIgD4AAJBBOJNYkEpApzCAQQAAkDkgAJBKQACQTEQAkDQohkiATAAAgDQAhkiASgCGSJA9LJNYgEoAAJBDNACQTECnMJBNPACQMiSTWIBDAACATACTWIA9AACATQAAgDIAAJA+LJNYgDkAk1iQQSyTWJBNNKBokDkkAJBOOACQMCSaIIA+AACQPiSTWIBBAACATQCGSIA5AACATgAAgDAAjRCAPgAAkEIkk1iQTiyTWIBCAACQTiynMJBPMACQLxgAkDcYrXiATgCGSIBOAIZIgE8AAJA+JKBokEEkhkiALwAAgDcAk1iQQzynMJBHNACQMhwAkEVAk1iQNxyTWIA+AACARwAAgDIAAJA7LACQQUCNEIBBAIZIgEMAoGiQQ0iaIIBFAACANwAAgDsAAIBBAACQSEgAkDAkAJA3HKBogEMAhkiASAAAgDAAAJA8MJNYgDcAAJBAOACQSEi7CJBDSACQKCiNEJAwKIZIgEAAAIBIAIZIgDwAjRCQNygAkDxAjRCAQwAAgCgAmiCQQ0iTWIA3AACAPAAAkEVIAJApOJNYgEMAk1iARQAAgCkAAJA5OJNYkDw8AJBFSKcwgDAAAIA8AACARQAAkEFEAJAmNACQMCiTWIA5AIZIgEEAAIAmAACQNTAAkDw8miCAMACGSJBBRKcwkD5IAJArMACQMCiGSIA1AACAPACNEIBBAI0QkDUshkiQPEAAkD5IjRCAPgAAgCsAAIAwAIZIgDUApzCQKygAkENIAJAvLJNYgDwAAIA+AJNYgCsAAIBDAACALwAAkDUwmiCANQAAkDs4AJBDRLRAgDsAAIBDAACQQEAAkCQsAJArLACQOzgAkENEmiCAOwAAgEMAjRCAQAAAgCQAAIArAACQNCyTWJA8PACQQ0yGSIA0ALRAkEdEAJAyKI0QkEVEhkiQNyiTWIA8AACAQwCNEIBHAACAMgCGSJA7NJNYkEE8AJBDTKBogEUAhkiQMDAAkEhQjRCAQQAAgEMAhkiAOwCTWIA3AACAMAAAgEgAAJA8OACQQECnMIA8AACAQAAAkEhQAJA8OACQQECaIJBDRACQKDSNEIA8AACAQAAAkDAwpzCASAAAkDcwhkiAQwAAgCgAk1iQPDgAkENEtECQRUAAkCk4miCAPAAAgEMAjRCANwCNEIBFAACAKQAAkDkshkiQPDiNEIA5AIZIkEU8pzCQQTwAkCY4k1iAPAAAgEUAAJA1NJNYgDAAAIBBAACAJgAAkDw0AJBBPIZIgDUArXiQPjwAkCssAJAwLIZIgDwAAIBBAJNYkDUomiCAPgAAgCsAAIAwAACANQAAkDwwjRCQPjinMJBDNACQKygAkC8kpzCAPgAAkDUkhkiAPACGSIBDAACAKwAAgC8AmiCQOyzOYIA1AACQNSQAkEMsmiCQPCQAkCQgAJArGACQNCCnMIA1AJNYgEMAAIA8AACAJAAAgCsAk1iANACaIIA7AIZIkEdIyBiQSkQAkEhMmiCQNCAAkDsck1iARwCGSIBKAI0QgDQAAJBANJNYkENEjRCASACGSIA7AACAQAAAkEdQAJBDRACQOxynMJBOTACQMzCnMIBDAACARwAAgEMAAJBCOACQRUSNEIBOAACAMwCNEIA7AI0QkEdUk1iQSlAAkDQsk1iAQgAAgEUAAJA7LACQNCwAkEhQjRCARwCGSIBKAACANAAAkEBAk1iAOwAAgDQAAIBAAACQQ0wAkEdQuwiAQwAAgEcAAJBMTACQMigAkDcshkiASACaIJA7OACQQESGSIBMAACAMgAAgDcAk1iAOwAAgEAAAJBDUACQOzgAkEBEmiCQR0SNEJBFRACQMSQAkDckjRCAOwAAgEAAhkiAQwCGSIBHAI0QgDEAAJA5MJNYgDkAAJBAOJNYkENEk1iARQAAkEdAAJAyJK14kDwshkiARwAAgDIAhkiANwAAgEAAAIBDAI0QkD48AJBHPI0QgDwAoGiQRTwAkDIkAJA2KI0QgD4AAIBHAIZIkDwok1iARQAAgDIAAIA2AJNYkEU0AJA+NKcwkEM4AJArIACQMiSnMIBFAACAPgAAgEMAAIArAACAMgAAkDssk1iAPACGSIA7AACQPjQAkEdEtECQSkAAkDQkk1iQSEAAkDskhkiAPgAAgEcAjRCASgAAgDQAjRCQQCyaIJBDMJNYgEgAhkiQR0C0QIA7AACAQAAAgEMAAJBDMACQR0AAkDskAJBOQACQMyCTWIBDAIZIgEcAAJBCNACQRUCGSIBHAIZIgDsAAIBOAACAMwCTWJBHTJNYkEpEAJA0KJNYkDssAJBISIZIgEIAAIBFAI0QgEcAAIBKAACANAAAkEA0k1iQQzgAkEdEjRCAOwCteIBDAACARwAAkExAAJAyJACQNyiGSIBIAI0QgEAAAJA7LJNYgEwAAIAyAACANwAAkEA4AJBDSLRAkEdAmiCAOwAAgEAAAIBDAACQMSQAkDckAJBHQACQRTyNEIBHAIZIkDkoAJBANJNYgEcAjRCAMQCGSJBDSJNYgEUAk1iQR0AAkDIgk1iAOQAAgEAAk1iAQwAAgEcAAIAyAACQPCgAkD5ApzCANwAAkEdEk1iQRUAAkDIkAJA2KJNYgDwAAIA+AJNYgEcAAIBFAACAMgAAgDYAAJA8LKcwkD48pzCAPAAAkD48AJBFQJogkENAAJArIACQMiCgaIBFAJNYgD4AAIA+AACAQwAAgCsAAIAyAACQOygAkD5ApzCQQ0inMJBHOJNYkEU8hkiQNyCNEJAyHI0QgDsAAIA+AIZIgEcAk1iAQwAAkDsok1iAMgCGSJBBMLRAgEUAAIA3AACAOwAAkEg4AJAwIACQQTAAkENAAJBFPACQNyCTWIBFAKBogEEAAIBDAACQPCiGSIBIAACAMACGSIA3AI0QgEEAAIA8AACQQDyTWJBDTJNYkEdAjRCQRUSGSJAyJACQNyCTWIBHAACQOzSTWIBAAACAQwAAgDIAhkiAOwAAkEFAmiCQQ0yaIJBISACQMCyNEIBFAI0QgEEAhkiAQwCGSIA3AACASAAAgDAAAJA8MJNYkEA8AJBIRJNYgDwAAJBAPACQSESaIJBJTACQNyyGSIBAAACASACGSJA5IJNYgEAAAIBIAACQPTQAkEBAhkiASQAAgDcAoGiQSUyTWIA9AACAQAAAkEpQAJA1MKBogEkAhkiASgAAgDUAAJA+OACQQUCNEIA5AI0QkEpMmiCQTEwAkDQohkiQOSyTWIA+AACAQQAAgEwAAIA0AIZIgEoAjRCQPTSGSJBDQACQTEyNEIA9AKcwgDkAAJBNSACQMigAkENAAJBMTJoggEMAAIBMAJoggE0AAIAyAACQPjAAkEE4miCAQwAAgEwAk1iQRUigaJBIRACQNyCGSJBGQACQOSSgaIA+AACAQQAAgEgAAIA3AI0QkD0sjRCARQAAkEA0AJBFQJoggD0AjRCQSkgAkDUsk1iARgAAgEAAAIBFAJNYgDkAAIBKAACANQAAkD4wjRCQQTwAkEVIjRCAPgCNEIBBAACARQAAkEE8AJBFSJNYkEhAAJA3KIZIkEZEjRCQOSCNEIBBAACARQCGSIBIAACANwCTWJA9KJNYkEA0AJBFRIZIgD0AoGiQSkgAkDUohkiARgCGSIBAAACARQCaIIBKAACANQAAkD44AJBBQJNYgDkAjRCQSkyaIIA+AACAQQAAkExMAJA0LACQOSSaIIBKAACATAAAgDQAAJA9OACQQ0SgaJBMUJNYgDkAAIA9AACAQwAAkE1QAJAyKACQPTgAkENEAJA5JIZIgD0AAIBDAJoggEwAhkiATQAAgDIAAIA5AACQPjyNEJBBQACQTUy0QJBOUACQOSgAkDAkhkiAQQAAgE0AmiCQPjiGSIA+AACATgAAgDkAAIAwAI0QkEJIk1iAPgCGSJBOULRAkE9IAJAvKACQNySaIIBCAACATgCNEJA+KJNYgE8AAIAvAACQQSyTWIA3AACAPgAAkEM8pzCQRziNEJBFPIZIkDIgAJA3JJNYgEMAhkiAQQCGSIBHAIZIkDsojRCAMgCGSJBBLACQQ0CnMJBIQACQMCiNEIBFAACAQQAAgEMAk1iANwCGSIA7AACASAAAgDAAAJA8LJNYkEA0AJBIRKcwkENAAJAoKI0QgEAAAIBIAIZIgDwAAJAwLJNYgEMAAIAoAJNYkDcwAJA8NJNYkENIk1iANwAAgDwAAJBFSACQKTiTWJA5NI0QgEMAhkiARQAAgCkAk1iQPECGSJBFRJogkEFIAJAmNJoggDkAAIBFAACQNSyNEIBBAACAJgCGSIA8AIZIgDUAAJA8PACQQUigaIAwAJNYkD5IAJArLACQMCSGSIA8AACAQQCTWJA1LJoggD4AAIArAACAMAAAgDUAAJA8PACQPkSaIIA8AACAPgAAkDw8AJA+RJNYkENAAJArJACQLyiNEIA8AACAPgCTWJA1LIZIgEMAAIArAACALwCTWIA1AACQOzwAkENApzCQQDwAkCQwAJArLKcwgDsAAIBDAACAQAAAgCQAAIArAACQNDCaIIA0AACQPDgAkENIyBiQRziGSJBFQJoggDwAAIBDAJoggEcAAJA7MACQMigAkDcojRCAMgCGSIA7AACQQTgAkENApzCQSEAAkDAsk1iAQQAAgEMAmiCQPCiGSIBIAACAMACGSIA3AI0QkEAwhkiQSDSGSIBFAKcwkEM4AJAoJI0QkDAkk1iASACGSIA8AIZIgEAAhkiAQwAAgCgAAJA3KJNYkDwwAJBDPKcwkEU8AJApMI0QgDwAAIBDAIZIgDAAAIA3AACQOTAAkEU8AJApMJNYgEUAAIApAIZIkDw0AJBFPLRAkEFAAJAmMIZIgDwAAIBFAI0QgDkAhkiQNTCNEIBFAACAKQAAgEEAAIAmAI0QgDUAAJA8NACQQTiteJA+PACQKyQAkDAgoGiAPAAAgEEAhkiAPgAAgCsAAIAwAACQNSyTWJA8LACQPkCGSIA1ALRAkEM0AJArIACQLySTWIA8AACAPgAAgEMAAIArAACALwAAkEM0AJArIACQLySaIJA1KI0QgEMAAIArAACALwCTWJA7LJoggDUAoGiQQzCBxHCAOwAAkDssAJA8JACQKyAAkDQcAJAkJKBogDsAmiCAPAAAgCsAAIA0AACAJACBiWiAQwAAkEBYAJA0QKcwkENUAJA3QACQQVgAkDVMzmCAQAAAgDQAAIBDAACANwAAgEEAAIA1AACQQFgAkDRAAJBDVACQN0AAkEFYAJA1TJNYgEAAAIA0AACAQwAAgDcAAIBBAACANQCTWJA0KACQQDzBUJBEOACQLSQAkDQgjRCANAAAgEAAjRCANACteJA5KACQRUCGSIBEAACALQCNEJA8IKcwkEAwmiCQRkgAkDQgAJAsKI0QgDkAAIBFAACAPAAAgEAAjRCALACaIIBGAACANAAAkDs0AJBHTJNYkD5ApzCQQDCnMJBKSACQLSgAkDQshkiAOwAAgEcAhkiAQACGSIA+AJNYgDQApzCASgAAgC0AAJA8IIZIkEhEjRCQQCinMJBFKKcwgDwAAIBFAACQMhwAkDUkk1iAQACTWIBIAACQMhwAkDUkk1iQOyyaIIAyAACANQCgaJBBNKcwgDIAAIA1AACAQQCGSIA7AM5g/y8A\");\n",
       "            });\n",
       "        }\n",
       "        if (typeof require === 'undefined') {\n",
       "            setTimeout(midiPlayerDiv9268_play, 2000);\n",
       "        } else {\n",
       "            midiPlayerDiv9268_play();\n",
       "        }\n",
       "        </script>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# answer to exercise 14.1 \n",
    "train1, _ = train[0]\n",
    "file_path=\"files/train1.midi\"\n",
    "decode_midi(train1.cpu().numpy(), file_path=file_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "5ff9ea02",
   "metadata": {},
   "outputs": [],
   "source": [
    "from torch.utils.data import DataLoader\n",
    "\n",
    "batch_size=2\n",
    "trainloader=DataLoader(train,batch_size=batch_size,\n",
    "                       shuffle=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b02df292",
   "metadata": {},
   "source": [
    "# 3\tBuild a GPT to generate music\n",
    "# 3.1\tHyperparameters in the music Transformer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "de6cc779",
   "metadata": {},
   "outputs": [],
   "source": [
    "from torch import nn\n",
    "class Config():\n",
    "    def __init__(self):\n",
    "        self.n_layer = 6\n",
    "        self.n_head = 8\n",
    "        self.n_embd = 512\n",
    "        self.vocab_size = 390\n",
    "        self.block_size = 2048 \n",
    "        self.embd_pdrop = 0.1\n",
    "        self.resid_pdrop = 0.1\n",
    "        self.attn_pdrop = 0.1\n",
    "        \n",
    "# instantiate a Config() class\n",
    "config=Config()\n",
    "device=\"cuda\" if torch.cuda.is_available() else \"cpu\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "73036c27",
   "metadata": {},
   "source": [
    "## 3.2 Build the music Transformer"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "69bf11b7",
   "metadata": {},
   "source": [
    "```python\n",
    "# defined in ch14util.py, same as the one defined in ch12\n",
    "class GELU(nn.Module):\n",
    "    def forward(self, x):\n",
    "        return 0.5*x*(1.0+torch.tanh(math.sqrt(2.0/math.pi)*\\\n",
    "                       (x + 0.044715 * torch.pow(x, 3.0))))\n",
    "```    "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "259b6597",
   "metadata": {},
   "source": [
    "```python\n",
    "# defined in ch14util.py, same as the one defined in ch12\n",
    "class CausalSelfAttention(nn.Module):\n",
    "    def __init__(self, config):\n",
    "        super().__init__()\n",
    "        self.c_attn = nn.Linear(config.n_embd, 3 * config.n_embd)\n",
    "        self.c_proj = nn.Linear(config.n_embd, config.n_embd)\n",
    "        self.attn_dropout = nn.Dropout(config.attn_pdrop)\n",
    "        self.resid_dropout = nn.Dropout(config.resid_pdrop)\n",
    "        self.register_buffer(\"bias\", torch.tril(torch.ones(\\\n",
    "                   config.block_size, config.block_size))\n",
    "             .view(1, 1, config.block_size, config.block_size))\n",
    "        self.n_head = config.n_head\n",
    "        self.n_embd = config.n_embd\n",
    "        \n",
    "    def forward(self, x):\n",
    "        B, T, C = x.size() \n",
    "        q, k ,v  = self.c_attn(x).split(self.n_embd, dim=2)\n",
    "        hs = C // self.n_head\n",
    "        k = k.view(B, T, self.n_head, hs).transpose(1, 2) \n",
    "        q = q.view(B, T, self.n_head, hs).transpose(1, 2) \n",
    "        v = v.view(B, T, self.n_head, hs).transpose(1, 2) \n",
    "\n",
    "        att = (q @ k.transpose(-2, -1)) *\\\n",
    "            (1.0 / math.sqrt(k.size(-1)))\n",
    "        att = att.masked_fill(self.bias[:,:,:T,:T] == 0, \\\n",
    "                              float('-inf'))\n",
    "        att = F.softmax(att, dim=-1)\n",
    "        att = self.attn_dropout(att)\n",
    "        y = att @ v \n",
    "        y = y.transpose(1, 2).contiguous().view(B, T, C)\n",
    "        y = self.resid_dropout(self.c_proj(y))\n",
    "        return y\n",
    "```    "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "05b9bb7e",
   "metadata": {},
   "source": [
    "```python\n",
    "# defined in ch14util.py, same as the one defined in ch12\n",
    "class Block(nn.Module):\n",
    "    def __init__(self, config):\n",
    "        super().__init__()\n",
    "        self.ln_1 = nn.LayerNorm(config.n_embd)\n",
    "        self.attn = CausalSelfAttention(config)\n",
    "        self.ln_2 = nn.LayerNorm(config.n_embd)\n",
    "        self.mlp = nn.ModuleDict(dict(\n",
    "            c_fc   = nn.Linear(config.n_embd, 4 * config.n_embd),\n",
    "            c_proj = nn.Linear(4 * config.n_embd, config.n_embd),\n",
    "            act    = GELU(),\n",
    "            dropout = nn.Dropout(config.resid_pdrop),\n",
    "        ))\n",
    "        m = self.mlp\n",
    "        self.mlpf=lambda x:m.dropout(m.c_proj(m.act(m.c_fc(x)))) \n",
    "\n",
    "    def forward(self, x):\n",
    "        x = x + self.attn(self.ln_1(x))\n",
    "        x = x + self.mlpf(self.ln_2(x))\n",
    "        return x\n",
    "```    "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cf523d95",
   "metadata": {},
   "source": [
    "```python\n",
    "# defined in ch14util.py, same as the one defined in ch12\n",
    "class Model(nn.Module):\n",
    "    def __init__(self, config):\n",
    "        super().__init__()\n",
    "        self.block_size = config.block_size\n",
    "        self.transformer = nn.ModuleDict(dict(\n",
    "            wte = nn.Embedding(config.vocab_size, config.n_embd),\n",
    "            wpe = nn.Embedding(config.block_size, config.n_embd),\n",
    "            drop = nn.Dropout(config.embd_pdrop),\n",
    "            h = nn.ModuleList([Block(config) \n",
    "                               for _ in range(config.n_layer)]),   \n",
    "            ln_f = nn.LayerNorm(config.n_embd),))\n",
    "        self.lm_head = nn.Linear(config.n_embd,\n",
    "                                 config.vocab_size, bias=False)      \n",
    "        for pn, p in self.named_parameters():\n",
    "            if pn.endswith('c_proj.weight'):    \n",
    "                torch.nn.init.normal_(p, mean=0.0, \n",
    "                  std=0.02/math.sqrt(2 * config.n_layer))\n",
    "    def forward(self, idx, targets=None):\n",
    "        b, t = idx.size()\n",
    "        pos = torch.arange(0,t,dtype=torch.long).unsqueeze(0).to(device)\n",
    "        tok_emb = self.transformer.wte(idx) \n",
    "        pos_emb = self.transformer.wpe(pos) \n",
    "        x = self.transformer.drop(tok_emb + pos_emb)\n",
    "        for block in self.transformer.h:\n",
    "            x = block(x)\n",
    "        x = self.transformer.ln_f(x)\n",
    "        logits = self.lm_head(x)\n",
    "        return logits\n",
    "```   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "8f611561",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "number of parameters: 20.16M\n",
      "Model(\n",
      "  (transformer): ModuleDict(\n",
      "    (wte): Embedding(390, 512)\n",
      "    (wpe): Embedding(2048, 512)\n",
      "    (drop): Dropout(p=0.1, inplace=False)\n",
      "    (h): ModuleList(\n",
      "      (0-5): 6 x Block(\n",
      "        (ln_1): LayerNorm((512,), eps=1e-05, elementwise_affine=True)\n",
      "        (attn): CausalSelfAttention(\n",
      "          (c_attn): Linear(in_features=512, out_features=1536, bias=True)\n",
      "          (c_proj): Linear(in_features=512, out_features=512, bias=True)\n",
      "          (attn_dropout): Dropout(p=0.1, inplace=False)\n",
      "          (resid_dropout): Dropout(p=0.1, inplace=False)\n",
      "        )\n",
      "        (ln_2): LayerNorm((512,), eps=1e-05, elementwise_affine=True)\n",
      "        (mlp): ModuleDict(\n",
      "          (c_fc): Linear(in_features=512, out_features=2048, bias=True)\n",
      "          (c_proj): Linear(in_features=2048, out_features=512, bias=True)\n",
      "          (act): GELU()\n",
      "          (dropout): Dropout(p=0.1, inplace=False)\n",
      "        )\n",
      "      )\n",
      "    )\n",
      "    (ln_f): LayerNorm((512,), eps=1e-05, elementwise_affine=True)\n",
      "  )\n",
      "  (lm_head): Linear(in_features=512, out_features=390, bias=False)\n",
      ")\n"
     ]
    }
   ],
   "source": [
    "from utils.ch14util import Model\n",
    "\n",
    "model=Model(config)\n",
    "model.to(device)\n",
    "num=sum(p.numel() for p in model.transformer.parameters())\n",
    "print(\"number of parameters: %.2fM\" % (num/1e6,))\n",
    "print(model)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "39e27a43",
   "metadata": {},
   "source": [
    "# 4. Train and use the music Transformer\n",
    "\n",
    "## 4.1\tTrain the music Transformer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "78d56b7c",
   "metadata": {},
   "outputs": [],
   "source": [
    "lr=0.0001\n",
    "optimizer = torch.optim.Adam(model.parameters(), lr=lr) \n",
    "# ignore the padding index\n",
    "loss_func=torch.nn.CrossEntropyLoss(ignore_index=389)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "5f7a9a50",
   "metadata": {},
   "outputs": [],
   "source": [
    "model.train()  \n",
    "for i in range(1,101):\n",
    "    tloss = 0.\n",
    "    for idx, (x,y) in enumerate(trainloader):\n",
    "        x,y=x.to(device),y.to(device)\n",
    "        output = model(x)\n",
    "        loss=loss_func(output.view(-1,output.size(-1)),\n",
    "                           y.view(-1))\n",
    "        optimizer.zero_grad()\n",
    "        loss.backward()\n",
    "        nn.utils.clip_grad_norm_(model.parameters(),1)\n",
    "        optimizer.step()\n",
    "        tloss += loss.item()\n",
    "    print(f'epoch {i} loss {tloss/(idx+1)}') \n",
    "torch.save(model.state_dict(),f'files/musicTrans.pth') "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cb3f6f49",
   "metadata": {},
   "source": [
    "## 4.2 Music Generation with the trained Transformer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "3c691166",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<pretty_midi.pretty_midi.PrettyMIDI at 0x233cfcf7fd0>"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from utils.processor import decode_midi\n",
    "\n",
    "prompt, _  = test[42]\n",
    "prompt = prompt.to(device)\n",
    "len_prompt=250\n",
    "\n",
    "file_path = \"files/prompt.midi\"\n",
    "decode_midi(prompt[:len_prompt].cpu().numpy(),\n",
    "            file_path=file_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "35dd928b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# answer to exercise 14.2\n",
    "prompt, _  = test[1]\n",
    "prompt = prompt.to(device)\n",
    "len_prompt=250\n",
    "file_path = \"files/prompt2.midi\"\n",
    "decode_midi(prompt[:len_prompt].cpu().numpy(),\n",
    "            file_path=file_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "ae48ec65",
   "metadata": {},
   "outputs": [],
   "source": [
    "# define the softmax function for later use\n",
    "softmax=torch.nn.Softmax(dim=-1)\n",
    "def sample(prompt,seq_length=1000,temperature=1):\n",
    "    # create input to feed to the transformer\n",
    "    gen_seq=torch.full((1,seq_length),389,dtype=torch.long).to(device)\n",
    "    idx=len(prompt)\n",
    "    gen_seq[..., :idx]=prompt.type(torch.long).to(device)\n",
    "    while(idx < seq_length):\n",
    "        y=softmax(model(gen_seq[..., :idx])/temperature)[...,:388]\n",
    "        probs=y[:, idx-1, :]\n",
    "        distrib=torch.distributions.categorical.Categorical(probs=probs)\n",
    "        next_token=distrib.sample()\n",
    "        gen_seq[:, idx]=next_token\n",
    "        idx+=1\n",
    "    return gen_seq[:, :idx]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "1d1c647a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Model(\n",
       "  (transformer): ModuleDict(\n",
       "    (wte): Embedding(390, 512)\n",
       "    (wpe): Embedding(2048, 512)\n",
       "    (drop): Dropout(p=0.1, inplace=False)\n",
       "    (h): ModuleList(\n",
       "      (0-5): 6 x Block(\n",
       "        (ln_1): LayerNorm((512,), eps=1e-05, elementwise_affine=True)\n",
       "        (attn): CausalSelfAttention(\n",
       "          (c_attn): Linear(in_features=512, out_features=1536, bias=True)\n",
       "          (c_proj): Linear(in_features=512, out_features=512, bias=True)\n",
       "          (attn_dropout): Dropout(p=0.1, inplace=False)\n",
       "          (resid_dropout): Dropout(p=0.1, inplace=False)\n",
       "        )\n",
       "        (ln_2): LayerNorm((512,), eps=1e-05, elementwise_affine=True)\n",
       "        (mlp): ModuleDict(\n",
       "          (c_fc): Linear(in_features=512, out_features=2048, bias=True)\n",
       "          (c_proj): Linear(in_features=2048, out_features=512, bias=True)\n",
       "          (act): GELU()\n",
       "          (dropout): Dropout(p=0.1, inplace=False)\n",
       "        )\n",
       "      )\n",
       "    )\n",
       "    (ln_f): LayerNorm((512,), eps=1e-05, elementwise_affine=True)\n",
       "  )\n",
       "  (lm_head): Linear(in_features=512, out_features=390, bias=False)\n",
       ")"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.load_state_dict(torch.load(\"files/musicTrans.pth\"))\n",
    "model.eval()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "42e258fb",
   "metadata": {},
   "source": [
    "We then call the *sample()* function to generate music: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "9901a2b2",
   "metadata": {},
   "outputs": [],
   "source": [
    "from utils.processor import encode_midi\n",
    "\n",
    "file_path = \"files/prompt.midi\"\n",
    "prompt = torch.tensor(encode_midi(file_path))\n",
    "generated_music=sample(prompt, seq_length=1000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "0651a2cb",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "info removed pitch: 52\n",
      "info removed pitch: 83\n",
      "info removed pitch: 55\n",
      "info removed pitch: 68\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<pretty_midi.pretty_midi.PrettyMIDI at 0x233cfda4fd0>"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "music_data = generated_music[0].cpu().numpy()\n",
    "file_path = 'files/musicTrans.midi'\n",
    "decode_midi(music_data, file_path=file_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "4f5919ef",
   "metadata": {},
   "outputs": [],
   "source": [
    "# answer to exercise 14.3\n",
    "file_path = \"files/prompt2.midi\"\n",
    "prompt = torch.tensor(encode_midi(file_path))\n",
    "generated_music=sample(prompt, seq_length=1200,temperature=1)\n",
    "music_data = generated_music[0].cpu().numpy()\n",
    "file_path = 'files/musicTrans2.midi'\n",
    "decode_midi(music_data, file_path=file_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "5b904ef9",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "info removed pitch: 46\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<pretty_midi.pretty_midi.PrettyMIDI at 0x233cfdafad0>"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "file_path = \"files/prompt.midi\"\n",
    "prompt = torch.tensor(encode_midi(file_path))\n",
    "generated_music=sample(prompt, seq_length=1000,temperature=1.5)\n",
    "music_data = generated_music[0].cpu().numpy()\n",
    "file_path = 'files/musicHiTemp.midi'\n",
    "decode_midi(music_data, file_path=file_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "fd0a0c82",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<pretty_midi.pretty_midi.PrettyMIDI at 0x233cfda0250>"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# answer to exercise 14.4\n",
    "file_path = \"files/prompt.midi\"\n",
    "prompt = torch.tensor(encode_midi(file_path))\n",
    "generated_music=sample(prompt, seq_length=1000,temperature=0.7)\n",
    "music_data = generated_music[0].cpu().numpy()\n",
    "file_path = 'files/musicLowTemp.midi'\n",
    "decode_midi(music_data, file_path=file_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "38a679fb",
   "metadata": {},
   "source": [
    "You can listen to the music by pressing the play button below:\n",
    "\n",
    "https://gattonweb.uky.edu/faculty/lium/ml/musicTrans.mp3"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
